1607449459846.jpg

This is a project I have done in Artificial Intelligence and Deeo Learning center of iran. A MLP classifier reunnig on MNIST-FASHION dataset.

I downloaded the mnist_fashion dataset manually and it contains 4 files which are test، test_label, train and train_label.

Using id2xnumpy to open the files and output is numpy array.

In [1]:
import numpy as np
import idx2numpy

pre_X_train = idx2numpy.convert_from_file('D:/Internship/train-images-idx3-ubyte')
pre_Y_train = idx2numpy.convert_from_file('D:/Internship/train-labels-idx1-ubyte')
pre_X_test = idx2numpy.convert_from_file('D:/AInternship/t10k-images-idx3-ubyte')
pre_Y_test = idx2numpy.convert_from_file('D:/Internship/t10k-labels-idx1-ubyte')

02.png

Using matplotlib to show 80 samples of the train data set.

In [2]:
'''
from sklearn.model_selction import KFold

X = np.array ([1,2],[3,4],[1,2],[3,4])
Y = np.array ([1,2,3,4])
kf = KFold(n_splite = 2)

for train_index,test_index in kf.split(X):
    X_train , X_test = X[train_index] , X[test_index]
    Y_train , Y_test = Y[train_index] , Y[test_index]
'''
Out[2]:
'\nfrom sklearn.model_selction import KFold\n\nX = np.array ([1,2],[3,4],[1,2],[3,4])\nY = np.array ([1,2,3,4])\nkf = KFold(n_splite = 2)\n\nfor train_index,test_index in kf.split(X):\n    X_train , X_test = X[train_index] , X[test_index]\n    Y_train , Y_test = Y[train_index] , Y[test_index]\n'

03.png

Using matplotlib to show 80 samples of the train data set.

In [3]:
import matplotlib.pyplot as plt

rand = np.random.randint(0,59999,(1,80))

plt.figure ( figsize = (12,15) )

for i in range(80):
    Sample = pre_X_train[rand[0][i]]
    plt.subplot(10,10,i+1)
    plt.imshow( Sample , cmap = "binary" )
    plt.axis("off")
    if pre_Y_train[rand[0][i]] == 0 :
        plt.title('T-shirt/top', fontsize = 12)
    elif pre_Y_train[rand[0][i]] == 1 :
        plt.title('Trouser', fontsize = 12)
    elif pre_Y_train[rand[0][i]] == 2 :
        plt.title('Pullover', fontsize = 12)
    elif pre_Y_train[rand[0][i]] == 3 :
        plt.title('Dress', fontsize = 12)
    elif pre_Y_train[rand[0][i]] == 4 :
        plt.title('Coat', fontsize = 12)
    elif pre_Y_train[rand[0][i]] == 5 :
        plt.title('Sandal', fontsize = 12)
    elif pre_Y_train[rand[0][i]] == 6 :
        plt.title('Shirt', fontsize = 12)
    elif pre_Y_train[rand[0][i]] == 7 :
        plt.title('Sneaker', fontsize = 12)
    elif pre_Y_train[rand[0][i]] == 8 :
        plt.title('Bag', fontsize = 12)
    elif pre_Y_train[rand[0][i]] == 9 :
        plt.title('Ankle boot', fontsize = 12)
    plt.show

Using matplotlib to show 80 samples of the test data.

In [4]:
import matplotlib.pyplot as plt

rand = np.random.randint(0,9999,(1,80))

plt.figure ( figsize = (12,15) )

for i in range(80):
    Sample = pre_X_test[rand[0][i]]
    plt.subplot(10,10,i+1)
    plt.imshow( Sample , cmap = "binary" )
    plt.axis("off")
    if pre_Y_test[rand[0][i]] == 0 :
        plt.title('T-shirt/top', fontsize = 12)
    elif pre_Y_test[rand[0][i]] == 1 :
        plt.title('Trouser', fontsize = 12)
    elif pre_Y_test[rand[0][i]] == 2 :
        plt.title('Pullover', fontsize = 12)
    elif pre_Y_test[rand[0][i]] == 3 :
        plt.title('Dress', fontsize = 12)
    elif pre_Y_test[rand[0][i]] == 4 :
        plt.title('Coat', fontsize = 12)
    elif pre_Y_test[rand[0][i]] == 5 :
        plt.title('Sandal', fontsize = 12)
    elif pre_Y_test[rand[0][i]] == 6 :
        plt.title('Shirt', fontsize = 12)
    elif pre_Y_test[rand[0][i]] == 7 :
        plt.title('Sneaker', fontsize = 12)
    elif pre_Y_test[rand[0][i]] == 8 :
        plt.title('Bag', fontsize = 12)
    elif pre_Y_test[rand[0][i]] == 9 :
        plt.title('Ankle boot', fontsize = 12)
    plt.show

04.png

Considering the fact that the elements of numy arrays are between 0 and 255, I divide the elements by 255 to normalize.

In [5]:
X_train = pre_X_train.reshape(60000, 784)
X_test = pre_X_test.reshape(10000, 784)

X_train = X_train.astype('float32')
X_test = X_test.astype('float32')

X_train /= 255
X_test /= 255

from keras.utils import np_utils
Y_train = np_utils.to_categorical(pre_Y_train)
Y_test = np_utils.to_categorical(pre_Y_test)

05.png

difinite the model. the number of neuron in evey layer will change to observe the effect. first layer=200 second = 50 third = 10

In [6]:
from keras.models import Sequential
from keras.layers import Dense
from keras.optimizers import SGD
from keras.losses import categorical_crossentropy

MLP_Model = Sequential()
MLP_Model.add(Dense(200, activation='relu', input_shape=(784,)))
MLP_Model.add(Dense(50, activation='relu'))
MLP_Model.add(Dense(10, activation='softmax'))

MLP_Model.summary()
MLP_Model.compile(optimizer = SGD(lr=0.001) , loss = categorical_crossentropy , metrics=['accuracy'])
Model: "sequential"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 dense (Dense)               (None, 200)               157000    
                                                                 
 dense_1 (Dense)             (None, 50)                10050     
                                                                 
 dense_2 (Dense)             (None, 10)                510       
                                                                 
=================================================================
Total params: 167,560
Trainable params: 167,560
Non-trainable params: 0
_________________________________________________________________
C:\Users\ali\anaconda3\lib\site-packages\keras\optimizers\optimizer_v2\gradient_descent.py:108: UserWarning: The `lr` argument is deprecated, use `learning_rate` instead.
  super(SGD, self).__init__(name, **kwargs)

06.png

fit X_train and Y_train using MLP_Model.fit() with epoch=25. usinf history in order to show loss, val_loss, accuracy and val_accuracy of each epoch.

In [7]:
network_history = MLP_Model.fit(X_train, Y_train, batch_size=128, epochs=25, validation_split=0.2)

history = network_history.history
import matplotlib.pyplot as plt
losses = history['loss']
val_losses = history['val_loss']
accuracies = history['accuracy']
val_accuracies = history['val_accuracy']
Epoch 1/25
375/375 [==============================] - 2s 4ms/step - loss: 2.1369 - accuracy: 0.3178 - val_loss: 1.9365 - val_accuracy: 0.4450
Epoch 2/25
375/375 [==============================] - 1s 3ms/step - loss: 1.7799 - accuracy: 0.5194 - val_loss: 1.6209 - val_accuracy: 0.5842
Epoch 3/25
375/375 [==============================] - 1s 3ms/step - loss: 1.4911 - accuracy: 0.6158 - val_loss: 1.3611 - val_accuracy: 0.6454
Epoch 4/25
375/375 [==============================] - 1s 3ms/step - loss: 1.2723 - accuracy: 0.6556 - val_loss: 1.1788 - val_accuracy: 0.6687
Epoch 5/25
375/375 [==============================] - 1s 3ms/step - loss: 1.1210 - accuracy: 0.6769 - val_loss: 1.0532 - val_accuracy: 0.6881
Epoch 6/25
375/375 [==============================] - 1s 3ms/step - loss: 1.0169 - accuracy: 0.6952 - val_loss: 0.9661 - val_accuracy: 0.7063
Epoch 7/25
375/375 [==============================] - 1s 3ms/step - loss: 0.9425 - accuracy: 0.7124 - val_loss: 0.9029 - val_accuracy: 0.7242
Epoch 8/25
375/375 [==============================] - 1s 3ms/step - loss: 0.8874 - accuracy: 0.7257 - val_loss: 0.8550 - val_accuracy: 0.7337
Epoch 9/25
375/375 [==============================] - 1s 3ms/step - loss: 0.8446 - accuracy: 0.7367 - val_loss: 0.8174 - val_accuracy: 0.7418
Epoch 10/25
375/375 [==============================] - 1s 4ms/step - loss: 0.8101 - accuracy: 0.7449 - val_loss: 0.7871 - val_accuracy: 0.7502
Epoch 11/25
375/375 [==============================] - 1s 3ms/step - loss: 0.7814 - accuracy: 0.7523 - val_loss: 0.7613 - val_accuracy: 0.7555
Epoch 12/25
375/375 [==============================] - 1s 3ms/step - loss: 0.7571 - accuracy: 0.7575 - val_loss: 0.7396 - val_accuracy: 0.7623
Epoch 13/25
375/375 [==============================] - 1s 3ms/step - loss: 0.7361 - accuracy: 0.7641 - val_loss: 0.7205 - val_accuracy: 0.7666
Epoch 14/25
375/375 [==============================] - 1s 3ms/step - loss: 0.7178 - accuracy: 0.7703 - val_loss: 0.7037 - val_accuracy: 0.7708
Epoch 15/25
375/375 [==============================] - 1s 3ms/step - loss: 0.7015 - accuracy: 0.7756 - val_loss: 0.6892 - val_accuracy: 0.7761
Epoch 16/25
375/375 [==============================] - 1s 3ms/step - loss: 0.6867 - accuracy: 0.7789 - val_loss: 0.6760 - val_accuracy: 0.7795
Epoch 17/25
375/375 [==============================] - 1s 3ms/step - loss: 0.6736 - accuracy: 0.7825 - val_loss: 0.6637 - val_accuracy: 0.7837
Epoch 18/25
375/375 [==============================] - 1s 3ms/step - loss: 0.6616 - accuracy: 0.7853 - val_loss: 0.6533 - val_accuracy: 0.7847
Epoch 19/25
375/375 [==============================] - 1s 3ms/step - loss: 0.6507 - accuracy: 0.7885 - val_loss: 0.6428 - val_accuracy: 0.7889
Epoch 20/25
375/375 [==============================] - 1s 3ms/step - loss: 0.6404 - accuracy: 0.7921 - val_loss: 0.6341 - val_accuracy: 0.7902
Epoch 21/25
375/375 [==============================] - 1s 3ms/step - loss: 0.6311 - accuracy: 0.7944 - val_loss: 0.6251 - val_accuracy: 0.7948
Epoch 22/25
375/375 [==============================] - 1s 3ms/step - loss: 0.6224 - accuracy: 0.7979 - val_loss: 0.6175 - val_accuracy: 0.7961
Epoch 23/25
375/375 [==============================] - 1s 3ms/step - loss: 0.6144 - accuracy: 0.7999 - val_loss: 0.6098 - val_accuracy: 0.7993
Epoch 24/25
375/375 [==============================] - 1s 3ms/step - loss: 0.6068 - accuracy: 0.8020 - val_loss: 0.6029 - val_accuracy: 0.8019
Epoch 25/25
375/375 [==============================] - 1s 3ms/step - loss: 0.5997 - accuracy: 0.8045 - val_loss: 0.5959 - val_accuracy: 0.8041

show the fact that loss and val_loss decrease by each epoch

In [8]:
plt.xlabel('Epochs')
plt.ylabel('Loss')
plt.plot(losses)
plt.plot(val_losses)
plt.legend(['loss', 'val_loss'])
Out[8]:
<matplotlib.legend.Legend at 0x25cca74b8c8>

show the fact that accuray and val_accuray increase by each epoch.

In [9]:
plt.xlabel('Epochs')
plt.ylabel('Accuracy')
plt.plot(accuracies)
plt.plot(val_accuracies)
plt.legend(['acc', 'val_acc'])
Out[9]:
<matplotlib.legend.Legend at 0x25cd37559c8>

run the model on test data and evaluate the loss and accuracy

In [10]:
test_loss, test_acc = MLP_Model.evaluate(X_test, Y_test)

Y_test_predict = MLP_Model.predict(X_test)

Y_test_predict = np.argmax(Y_test_predict, axis=1)
313/313 [==============================] - 0s 1ms/step - loss: 0.6177 - accuracy: 0.7912
313/313 [==============================] - 0s 1ms/step

show 80 samples of predicted data.

In [11]:
import matplotlib.pyplot as plt

rand = np.random.randint(0,9999,(1,80))

plt.figure ( figsize = (12,15) )

for i in range(80):
    Sample = pre_X_test[rand[0][i]]
    plt.subplot(10,10,i+1)
    plt.imshow( Sample , cmap = "binary" )
    plt.axis("off")
    if Y_test_predict[rand[0][i]] == 0 :
        plt.title('T-shirt/top', fontsize = 12)
    elif Y_test_predict[rand[0][i]] == 1 :
        plt.title('Trouser', fontsize = 12)
    elif Y_test_predict[rand[0][i]] == 2 :
        plt.title('Pullover', fontsize = 12)
    elif Y_test_predict[rand[0][i]] == 3 :
        plt.title('Dress', fontsize = 12)
    elif Y_test_predict[rand[0][i]] == 4 :
        plt.title('Coat', fontsize = 12)
    elif Y_test_predict[rand[0][i]] == 5 :
        plt.title('Sandal', fontsize = 12)
    elif Y_test_predict[rand[0][i]] == 6 :
        plt.title('Shirt', fontsize = 12)
    elif Y_test_predict[rand[0][i]] == 7 :
        plt.title('Sneaker', fontsize = 12)
    elif Y_test_predict[rand[0][i]] == 8 :
        plt.title('Bag', fontsize = 12)
    elif Y_test_predict[rand[0][i]] == 9 :
        plt.title('Ankle boot', fontsize = 12)
    plt.show

calculation of confusion matrix

In [12]:
from sklearn.metrics import confusion_matrix

#Y_test_predict = Y_test_predict.astype('float32')

confusion_matrix ( pre_Y_test , Y_test_predict )
Out[12]:
array([[785,   4,  15,  77,   6,   1,  94,   0,  18,   0],
       [  6, 927,  15,  43,   7,   0,   0,   0,   2,   0],
       [ 20,   1, 658,  11, 189,   1, 105,   0,  15,   0],
       [ 39,  19,   6, 827,  36,   1,  66,   0,   6,   0],
       [  1,   2, 119,  48, 734,   0,  87,   0,   9,   0],
       [  1,   0,   0,   2,   0, 812,   0, 108,   7,  70],
       [189,   3, 140,  42, 161,   1, 426,   0,  38,   0],
       [  0,   0,   0,   0,   0,  28,   0, 892,   0,  80],
       [  2,   1,  13,  15,   3,  10,  21,   8, 925,   2],
       [  1,   0,   0,   1,   0,  11,   0,  60,   1, 926]], dtype=int64)

07.png

change the number of neurons(200 to 1000 in first layer) and observe the effect.

In [13]:
from keras.models import Sequential
from keras.layers import Dense
from keras.optimizers import SGD
from keras.losses import categorical_crossentropy

MLP_Model = Sequential()
MLP_Model.add(Dense(1000, activation='relu', input_shape=(784,)))
MLP_Model.add(Dense(100, activation='relu'))
MLP_Model.add(Dense(10, activation='softmax'))

MLP_Model.summary()
MLP_Model.compile(optimizer = SGD(lr=0.001) , loss = categorical_crossentropy , metrics=['accuracy'])
Model: "sequential_1"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 dense_3 (Dense)             (None, 1000)              785000    
                                                                 
 dense_4 (Dense)             (None, 100)               100100    
                                                                 
 dense_5 (Dense)             (None, 10)                1010      
                                                                 
=================================================================
Total params: 886,110
Trainable params: 886,110
Non-trainable params: 0
_________________________________________________________________
C:\Users\ali\anaconda3\lib\site-packages\keras\optimizers\optimizer_v2\gradient_descent.py:108: UserWarning: The `lr` argument is deprecated, use `learning_rate` instead.
  super(SGD, self).__init__(name, **kwargs)

observe the effect of the number of neurons on losses, val_losses, accuracy and val_accuracy.

In [14]:
network_history = MLP_Model.fit(X_train, Y_train, batch_size=128, epochs=25, validation_split=0.2)

history = network_history.history
import matplotlib.pyplot as plt
losses = history['loss']
val_losses = history['val_loss']
accuracies = history['accuracy']
val_accuracies = history['val_accuracy']
Epoch 1/25
375/375 [==============================] - 3s 8ms/step - loss: 1.8860 - accuracy: 0.4576 - val_loss: 1.5802 - val_accuracy: 0.6417
Epoch 2/25
375/375 [==============================] - 3s 7ms/step - loss: 1.4016 - accuracy: 0.6641 - val_loss: 1.2392 - val_accuracy: 0.6848
Epoch 3/25
375/375 [==============================] - 3s 7ms/step - loss: 1.1468 - accuracy: 0.6886 - val_loss: 1.0518 - val_accuracy: 0.7029
Epoch 4/25
375/375 [==============================] - 3s 7ms/step - loss: 1.0007 - accuracy: 0.7076 - val_loss: 0.9390 - val_accuracy: 0.7213
Epoch 5/25
375/375 [==============================] - 2s 7ms/step - loss: 0.9091 - accuracy: 0.7251 - val_loss: 0.8663 - val_accuracy: 0.7397
Epoch 6/25
375/375 [==============================] - 3s 7ms/step - loss: 0.8468 - accuracy: 0.7388 - val_loss: 0.8139 - val_accuracy: 0.7483
Epoch 7/25
375/375 [==============================] - 3s 7ms/step - loss: 0.8009 - accuracy: 0.7514 - val_loss: 0.7742 - val_accuracy: 0.7606
Epoch 8/25
375/375 [==============================] - 3s 7ms/step - loss: 0.7649 - accuracy: 0.7606 - val_loss: 0.7427 - val_accuracy: 0.7684
Epoch 9/25
375/375 [==============================] - 3s 7ms/step - loss: 0.7355 - accuracy: 0.7689 - val_loss: 0.7163 - val_accuracy: 0.7724
Epoch 10/25
375/375 [==============================] - 3s 7ms/step - loss: 0.7109 - accuracy: 0.7751 - val_loss: 0.6940 - val_accuracy: 0.7800
Epoch 11/25
375/375 [==============================] - 2s 6ms/step - loss: 0.6898 - accuracy: 0.7821 - val_loss: 0.6754 - val_accuracy: 0.7854
Epoch 12/25
375/375 [==============================] - 3s 7ms/step - loss: 0.6714 - accuracy: 0.7875 - val_loss: 0.6582 - val_accuracy: 0.7910
Epoch 13/25
375/375 [==============================] - 2s 6ms/step - loss: 0.6552 - accuracy: 0.7922 - val_loss: 0.6437 - val_accuracy: 0.7954
Epoch 14/25
375/375 [==============================] - 2s 7ms/step - loss: 0.6407 - accuracy: 0.7963 - val_loss: 0.6315 - val_accuracy: 0.7982
Epoch 15/25
375/375 [==============================] - 2s 6ms/step - loss: 0.6277 - accuracy: 0.8002 - val_loss: 0.6191 - val_accuracy: 0.8018
Epoch 16/25
375/375 [==============================] - 3s 7ms/step - loss: 0.6161 - accuracy: 0.8037 - val_loss: 0.6081 - val_accuracy: 0.8041
Epoch 17/25
375/375 [==============================] - 3s 7ms/step - loss: 0.6055 - accuracy: 0.8065 - val_loss: 0.5982 - val_accuracy: 0.8075
Epoch 18/25
375/375 [==============================] - 2s 6ms/step - loss: 0.5958 - accuracy: 0.8091 - val_loss: 0.5897 - val_accuracy: 0.8088
Epoch 19/25
375/375 [==============================] - 2s 6ms/step - loss: 0.5868 - accuracy: 0.8114 - val_loss: 0.5810 - val_accuracy: 0.8112
Epoch 20/25
375/375 [==============================] - 3s 7ms/step - loss: 0.5787 - accuracy: 0.8142 - val_loss: 0.5733 - val_accuracy: 0.8144
Epoch 21/25
375/375 [==============================] - 2s 7ms/step - loss: 0.5710 - accuracy: 0.8155 - val_loss: 0.5668 - val_accuracy: 0.8151
Epoch 22/25
375/375 [==============================] - 3s 7ms/step - loss: 0.5639 - accuracy: 0.8185 - val_loss: 0.5599 - val_accuracy: 0.8176
Epoch 23/25
375/375 [==============================] - 3s 7ms/step - loss: 0.5574 - accuracy: 0.8195 - val_loss: 0.5544 - val_accuracy: 0.8182
Epoch 24/25
375/375 [==============================] - 3s 7ms/step - loss: 0.5513 - accuracy: 0.8209 - val_loss: 0.5484 - val_accuracy: 0.8196
Epoch 25/25
375/375 [==============================] - 3s 7ms/step - loss: 0.5456 - accuracy: 0.8224 - val_loss: 0.5431 - val_accuracy: 0.8223

show the fact that loss and val_loss decrease by each epoch

In [15]:
plt.xlabel('Epochs')
plt.ylabel('Loss')
plt.plot(losses)
plt.plot(val_losses)
plt.legend(['loss', 'val_loss'])
Out[15]:
<matplotlib.legend.Legend at 0x25cd668ce48>

show the fact that accuray and val_accuray increase by each epoch.

In [16]:
plt.xlabel('Epochs')
plt.ylabel('Accuracy')
plt.plot(accuracies)
plt.plot(val_accuracies)
plt.legend(['acc', 'val_acc'])
Out[16]:
<matplotlib.legend.Legend at 0x25cd670ab08>

run the model on test data and evaluate the loss and accuracy

In [17]:
test_loss, test_acc = MLP_Model.evaluate(X_test, Y_test)

Y_test_predict = MLP_Model.predict(X_test)

Y_test_predict = np.argmax(Y_test_predict, axis=1)
313/313 [==============================] - 1s 3ms/step - loss: 0.5644 - accuracy: 0.8098
313/313 [==============================] - 1s 3ms/step

show 80 samples of predicted data.

In [18]:
import matplotlib.pyplot as plt

rand = np.random.randint(0,9999,(1,80))

plt.figure ( figsize = (12,15) )

for i in range(80):
    Sample = pre_X_test[rand[0][i]]
    plt.subplot(10,10,i+1)
    plt.imshow( Sample , cmap = "binary" )
    plt.axis("off")
    if Y_test_predict[rand[0][i]] == 0 :
        plt.title('T-shirt/top', fontsize = 12)
    elif Y_test_predict[rand[0][i]] == 1 :
        plt.title('Trouser', fontsize = 12)
    elif Y_test_predict[rand[0][i]] == 2 :
        plt.title('Pullover', fontsize = 12)
    elif Y_test_predict[rand[0][i]] == 3 :
        plt.title('Dress', fontsize = 12)
    elif Y_test_predict[rand[0][i]] == 4 :
        plt.title('Coat', fontsize = 12)
    elif Y_test_predict[rand[0][i]] == 5 :
        plt.title('Sandal', fontsize = 12)
    elif Y_test_predict[rand[0][i]] == 6 :
        plt.title('Shirt', fontsize = 12)
    elif Y_test_predict[rand[0][i]] == 7 :
        plt.title('Sneaker', fontsize = 12)
    elif Y_test_predict[rand[0][i]] == 8 :
        plt.title('Bag', fontsize = 12)
    elif Y_test_predict[rand[0][i]] == 9 :
        plt.title('Ankle boot', fontsize = 12)
    plt.show

calculate the confusion matrix

In [19]:
from sklearn.metrics import confusion_matrix

#Y_test_predict = Y_test_predict.astype('float32')

confusion_matrix ( pre_Y_test , Y_test_predict )
Out[19]:
array([[807,   6,  13,  66,   5,   2,  80,   0,  21,   0],
       [  6, 931,  12,  39,   6,   0,   4,   0,   2,   0],
       [ 24,   2, 695,   9, 165,   1,  86,   0,  18,   0],
       [ 35,  14,  11, 856,  36,   1,  42,   0,   5,   0],
       [  0,   2, 113,  40, 763,   0,  74,   0,   8,   0],
       [  0,   0,   0,   2,   0, 853,   0,  83,   7,  55],
       [192,   3, 141,  46, 139,   0, 437,   0,  42,   0],
       [  0,   0,   0,   0,   0,  34,   0, 884,   0,  82],
       [  1,   1,  14,  10,   3,   7,  15,  10, 938,   1],
       [  0,   0,   0,   2,   0,   9,   0,  54,   1, 934]], dtype=int64)

change the the number of neurons in each layer.

In [20]:
from keras.models import Sequential
from keras.layers import Dense
from keras.optimizers import SGD
from keras.losses import categorical_crossentropy

MLP_Model = Sequential()
MLP_Model.add(Dense(500, activation='relu', input_shape=(784,)))
MLP_Model.add(Dense(250, activation='relu'))
MLP_Model.add(Dense(10, activation='softmax'))

MLP_Model.summary()
MLP_Model.compile(optimizer = SGD(lr=0.001) , loss = categorical_crossentropy , metrics=['accuracy'])
Model: "sequential_2"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 dense_6 (Dense)             (None, 500)               392500    
                                                                 
 dense_7 (Dense)             (None, 250)               125250    
                                                                 
 dense_8 (Dense)             (None, 10)                2510      
                                                                 
=================================================================
Total params: 520,260
Trainable params: 520,260
Non-trainable params: 0
_________________________________________________________________
C:\Users\ali\anaconda3\lib\site-packages\keras\optimizers\optimizer_v2\gradient_descent.py:108: UserWarning: The `lr` argument is deprecated, use `learning_rate` instead.
  super(SGD, self).__init__(name, **kwargs)

observe the effect of the number of neurons on losses, val_losses, accuracy and val_accuracy.

In [21]:
network_history = MLP_Model.fit(X_train, Y_train, batch_size=128, epochs=25, validation_split=0.2)

history = network_history.history
import matplotlib.pyplot as plt
losses = history['loss']
val_losses = history['val_loss']
accuracies = history['accuracy']
val_accuracies = history['val_accuracy']
Epoch 1/25
375/375 [==============================] - 2s 6ms/step - loss: 2.0006 - accuracy: 0.4082 - val_loss: 1.7202 - val_accuracy: 0.6055
Epoch 2/25
375/375 [==============================] - 2s 6ms/step - loss: 1.5287 - accuracy: 0.6483 - val_loss: 1.3500 - val_accuracy: 0.6794
Epoch 3/25
375/375 [==============================] - 2s 5ms/step - loss: 1.2405 - accuracy: 0.6798 - val_loss: 1.1295 - val_accuracy: 0.6952
Epoch 4/25
375/375 [==============================] - 2s 5ms/step - loss: 1.0673 - accuracy: 0.6990 - val_loss: 0.9953 - val_accuracy: 0.7127
Epoch 5/25
375/375 [==============================] - 2s 5ms/step - loss: 0.9582 - accuracy: 0.7165 - val_loss: 0.9064 - val_accuracy: 0.7243
Epoch 6/25
375/375 [==============================] - 2s 5ms/step - loss: 0.8842 - accuracy: 0.7311 - val_loss: 0.8455 - val_accuracy: 0.7424
Epoch 7/25
375/375 [==============================] - 2s 5ms/step - loss: 0.8307 - accuracy: 0.7442 - val_loss: 0.7991 - val_accuracy: 0.7552
Epoch 8/25
375/375 [==============================] - 2s 5ms/step - loss: 0.7899 - accuracy: 0.7545 - val_loss: 0.7630 - val_accuracy: 0.7626
Epoch 9/25
375/375 [==============================] - 2s 5ms/step - loss: 0.7573 - accuracy: 0.7628 - val_loss: 0.7344 - val_accuracy: 0.7692
Epoch 10/25
375/375 [==============================] - 2s 5ms/step - loss: 0.7304 - accuracy: 0.7700 - val_loss: 0.7104 - val_accuracy: 0.7756
Epoch 11/25
375/375 [==============================] - 2s 5ms/step - loss: 0.7077 - accuracy: 0.7766 - val_loss: 0.6894 - val_accuracy: 0.7816
Epoch 12/25
375/375 [==============================] - 2s 5ms/step - loss: 0.6880 - accuracy: 0.7822 - val_loss: 0.6717 - val_accuracy: 0.7861
Epoch 13/25
375/375 [==============================] - 2s 5ms/step - loss: 0.6706 - accuracy: 0.7875 - val_loss: 0.6558 - val_accuracy: 0.7918
Epoch 14/25
375/375 [==============================] - 2s 5ms/step - loss: 0.6553 - accuracy: 0.7926 - val_loss: 0.6423 - val_accuracy: 0.7940
Epoch 15/25
375/375 [==============================] - 2s 5ms/step - loss: 0.6415 - accuracy: 0.7960 - val_loss: 0.6298 - val_accuracy: 0.7993
Epoch 16/25
375/375 [==============================] - 2s 5ms/step - loss: 0.6292 - accuracy: 0.8007 - val_loss: 0.6193 - val_accuracy: 0.8023
Epoch 17/25
375/375 [==============================] - 2s 6ms/step - loss: 0.6180 - accuracy: 0.8032 - val_loss: 0.6082 - val_accuracy: 0.8028
Epoch 18/25
375/375 [==============================] - 2s 6ms/step - loss: 0.6078 - accuracy: 0.8066 - val_loss: 0.5984 - val_accuracy: 0.8059
Epoch 19/25
375/375 [==============================] - 2s 5ms/step - loss: 0.5984 - accuracy: 0.8090 - val_loss: 0.5897 - val_accuracy: 0.8092
Epoch 20/25
375/375 [==============================] - 2s 5ms/step - loss: 0.5898 - accuracy: 0.8106 - val_loss: 0.5821 - val_accuracy: 0.8117
Epoch 21/25
375/375 [==============================] - 2s 5ms/step - loss: 0.5817 - accuracy: 0.8128 - val_loss: 0.5753 - val_accuracy: 0.8136
Epoch 22/25
375/375 [==============================] - 2s 5ms/step - loss: 0.5743 - accuracy: 0.8149 - val_loss: 0.5681 - val_accuracy: 0.8157
Epoch 23/25
375/375 [==============================] - 2s 5ms/step - loss: 0.5675 - accuracy: 0.8163 - val_loss: 0.5615 - val_accuracy: 0.8164
Epoch 24/25
375/375 [==============================] - 2s 6ms/step - loss: 0.5611 - accuracy: 0.8179 - val_loss: 0.5567 - val_accuracy: 0.8180
Epoch 25/25
375/375 [==============================] - 2s 5ms/step - loss: 0.5551 - accuracy: 0.8197 - val_loss: 0.5505 - val_accuracy: 0.8202

show the fact that loss and val_loss decrease by each epoch

In [22]:
plt.xlabel('Epochs')
plt.ylabel('Loss')
plt.plot(losses)
plt.plot(val_losses)
plt.legend(['loss', 'val_loss'])
Out[22]:
<matplotlib.legend.Legend at 0x25cd40c1ec8>

show the fact that accuray and val_accuray increase by each epoch.

In [23]:
plt.xlabel('Epochs')
plt.ylabel('Accuracy')
plt.plot(accuracies)
plt.plot(val_accuracies)
plt.legend(['acc', 'val_acc'])
Out[23]:
<matplotlib.legend.Legend at 0x25cd4903388>

run the model on test data and evaluate the loss and accuracy

In [24]:
test_loss, test_acc = MLP_Model.evaluate(X_test, Y_test)

Y_test_predict = MLP_Model.predict(X_test)

Y_test_predict = np.argmax(Y_test_predict, axis=1)
313/313 [==============================] - 1s 2ms/step - loss: 0.5722 - accuracy: 0.8078
313/313 [==============================] - 1s 2ms/step

show 80 samples of predicted data.

In [25]:
import matplotlib.pyplot as plt

rand = np.random.randint(0,9999,(1,80))

plt.figure ( figsize = (12,15) )

for i in range(80):
    Sample = pre_X_test[rand[0][i]]
    plt.subplot(10,10,i+1)
    plt.imshow( Sample , cmap = "binary" )
    plt.axis("off")
    if Y_test_predict[rand[0][i]] == 0 :
        plt.title('T-shirt/top', fontsize = 12)
    elif Y_test_predict[rand[0][i]] == 1 :
        plt.title('Trouser', fontsize = 12)
    elif Y_test_predict[rand[0][i]] == 2 :
        plt.title('Pullover', fontsize = 12)
    elif Y_test_predict[rand[0][i]] == 3 :
        plt.title('Dress', fontsize = 12)
    elif Y_test_predict[rand[0][i]] == 4 :
        plt.title('Coat', fontsize = 12)
    elif Y_test_predict[rand[0][i]] == 5 :
        plt.title('Sandal', fontsize = 12)
    elif Y_test_predict[rand[0][i]] == 6 :
        plt.title('Shirt', fontsize = 12)
    elif Y_test_predict[rand[0][i]] == 7 :
        plt.title('Sneaker', fontsize = 12)
    elif Y_test_predict[rand[0][i]] == 8 :
        plt.title('Bag', fontsize = 12)
    elif Y_test_predict[rand[0][i]] == 9 :
        plt.title('Ankle boot', fontsize = 12)
    plt.show

calculate the confusion matrix

In [26]:
from sklearn.metrics import confusion_matrix

#Y_test_predict = Y_test_predict.astype('float32')

confusion_matrix ( pre_Y_test , Y_test_predict )
Out[26]:
array([[784,   7,   9,  67,   5,   2, 102,   0,  23,   1],
       [  3, 930,  15,  43,   6,   0,   1,   0,   2,   0],
       [ 21,   2, 682,   9, 164,   1, 107,   0,  13,   1],
       [ 29,  16,  10, 856,  31,   1,  52,   0,   5,   0],
       [  0,   2, 112,  39, 741,   0,  99,   0,   7,   0],
       [  1,   0,   0,   2,   0, 845,   0,  86,   9,  57],
       [163,   3, 131,  46, 126,   3, 490,   0,  38,   0],
       [  0,   0,   0,   0,   0,  39,   0, 883,   0,  78],
       [  1,   1,  11,  12,   2,   6,  24,   8, 934,   1],
       [  0,   0,   0,   2,   0,  13,   0,  51,   1, 933]], dtype=int64)

change the the number of neurons in each layer.

In [27]:
from keras.models import Sequential
from keras.layers import Dense
from keras.optimizers import SGD
from keras.losses import categorical_crossentropy

MLP_Model = Sequential()
MLP_Model.add(Dense(700, activation='relu', input_shape=(784,)))
MLP_Model.add(Dense(300, activation='relu'))
MLP_Model.add(Dense(10, activation='softmax'))

MLP_Model.summary()
MLP_Model.compile(optimizer = SGD(lr=0.001) , loss = categorical_crossentropy , metrics=['accuracy'])
Model: "sequential_3"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 dense_9 (Dense)             (None, 700)               549500    
                                                                 
 dense_10 (Dense)            (None, 300)               210300    
                                                                 
 dense_11 (Dense)            (None, 10)                3010      
                                                                 
=================================================================
Total params: 762,810
Trainable params: 762,810
Non-trainable params: 0
_________________________________________________________________
C:\Users\ali\anaconda3\lib\site-packages\keras\optimizers\optimizer_v2\gradient_descent.py:108: UserWarning: The `lr` argument is deprecated, use `learning_rate` instead.
  super(SGD, self).__init__(name, **kwargs)

observe the effect of the number of neurons on losses, val_losses, accuracy and val_accuracy.

In [28]:
network_history = MLP_Model.fit(X_train, Y_train, batch_size=128, epochs=25, validation_split=0.2)

history = network_history.history
import matplotlib.pyplot as plt
losses = history['loss']
val_losses = history['val_loss']
accuracies = history['accuracy']
val_accuracies = history['val_accuracy']
Epoch 1/25
375/375 [==============================] - 3s 7ms/step - loss: 1.9344 - accuracy: 0.4334 - val_loss: 1.6184 - val_accuracy: 0.6284
Epoch 2/25
375/375 [==============================] - 3s 7ms/step - loss: 1.4311 - accuracy: 0.6491 - val_loss: 1.2625 - val_accuracy: 0.6738
Epoch 3/25
375/375 [==============================] - 2s 7ms/step - loss: 1.1637 - accuracy: 0.6802 - val_loss: 1.0647 - val_accuracy: 0.6966
Epoch 4/25
375/375 [==============================] - 3s 7ms/step - loss: 1.0105 - accuracy: 0.7023 - val_loss: 0.9468 - val_accuracy: 0.7222
Epoch 5/25
375/375 [==============================] - 3s 7ms/step - loss: 0.9153 - accuracy: 0.7235 - val_loss: 0.8701 - val_accuracy: 0.7380
Epoch 6/25
375/375 [==============================] - 3s 7ms/step - loss: 0.8507 - accuracy: 0.7389 - val_loss: 0.8165 - val_accuracy: 0.7500
Epoch 7/25
375/375 [==============================] - 2s 6ms/step - loss: 0.8035 - accuracy: 0.7519 - val_loss: 0.7757 - val_accuracy: 0.7611
Epoch 8/25
375/375 [==============================] - 2s 7ms/step - loss: 0.7669 - accuracy: 0.7627 - val_loss: 0.7439 - val_accuracy: 0.7700
Epoch 9/25
375/375 [==============================] - 2s 7ms/step - loss: 0.7373 - accuracy: 0.7715 - val_loss: 0.7182 - val_accuracy: 0.7757
Epoch 10/25
375/375 [==============================] - 2s 6ms/step - loss: 0.7124 - accuracy: 0.7778 - val_loss: 0.6951 - val_accuracy: 0.7828
Epoch 11/25
375/375 [==============================] - 2s 6ms/step - loss: 0.6909 - accuracy: 0.7842 - val_loss: 0.6772 - val_accuracy: 0.7868
Epoch 12/25
375/375 [==============================] - 2s 7ms/step - loss: 0.6722 - accuracy: 0.7889 - val_loss: 0.6589 - val_accuracy: 0.7912
Epoch 13/25
375/375 [==============================] - 2s 6ms/step - loss: 0.6559 - accuracy: 0.7941 - val_loss: 0.6442 - val_accuracy: 0.7962
Epoch 14/25
375/375 [==============================] - 2s 6ms/step - loss: 0.6413 - accuracy: 0.7974 - val_loss: 0.6307 - val_accuracy: 0.7985
Epoch 15/25
375/375 [==============================] - 2s 7ms/step - loss: 0.6282 - accuracy: 0.8011 - val_loss: 0.6190 - val_accuracy: 0.8021
Epoch 16/25
375/375 [==============================] - 2s 6ms/step - loss: 0.6163 - accuracy: 0.8031 - val_loss: 0.6083 - val_accuracy: 0.8028
Epoch 17/25
375/375 [==============================] - 2s 7ms/step - loss: 0.6055 - accuracy: 0.8061 - val_loss: 0.5993 - val_accuracy: 0.8055
Epoch 18/25
375/375 [==============================] - 3s 7ms/step - loss: 0.5957 - accuracy: 0.8081 - val_loss: 0.5897 - val_accuracy: 0.8094
Epoch 19/25
375/375 [==============================] - 3s 7ms/step - loss: 0.5865 - accuracy: 0.8104 - val_loss: 0.5814 - val_accuracy: 0.8109
Epoch 20/25
375/375 [==============================] - 2s 6ms/step - loss: 0.5783 - accuracy: 0.8128 - val_loss: 0.5742 - val_accuracy: 0.8127
Epoch 21/25
375/375 [==============================] - 2s 6ms/step - loss: 0.5707 - accuracy: 0.8145 - val_loss: 0.5675 - val_accuracy: 0.8133
Epoch 22/25
375/375 [==============================] - 2s 7ms/step - loss: 0.5637 - accuracy: 0.8159 - val_loss: 0.5610 - val_accuracy: 0.8143
Epoch 23/25
375/375 [==============================] - 2s 7ms/step - loss: 0.5572 - accuracy: 0.8170 - val_loss: 0.5550 - val_accuracy: 0.8161
Epoch 24/25
375/375 [==============================] - 2s 7ms/step - loss: 0.5510 - accuracy: 0.8189 - val_loss: 0.5496 - val_accuracy: 0.8179
Epoch 25/25
375/375 [==============================] - 2s 7ms/step - loss: 0.5452 - accuracy: 0.8203 - val_loss: 0.5443 - val_accuracy: 0.8161

show the fact that loss and val_loss decrease by each epoch

In [29]:
plt.xlabel('Epochs')
plt.ylabel('Loss')
plt.plot(losses)
plt.plot(val_losses)
plt.legend(['loss', 'val_loss'])
Out[29]:
<matplotlib.legend.Legend at 0x25cd47e7a48>

show the fact that accuray and val_accuray increase by each epoch.

In [30]:
plt.xlabel('Epochs')
plt.ylabel('Accuracy')
plt.plot(accuracies)
plt.plot(val_accuracies)
plt.legend(['acc', 'val_acc'])
Out[30]:
<matplotlib.legend.Legend at 0x25cd4467148>

run the model on test data and evaluate the loss and accuracy

In [31]:
test_loss, test_acc = MLP_Model.evaluate(X_test, Y_test)

Y_test_predict = MLP_Model.predict(X_test)

Y_test_predict = np.argmax(Y_test_predict, axis=1)
313/313 [==============================] - 1s 3ms/step - loss: 0.5654 - accuracy: 0.8084
313/313 [==============================] - 1s 3ms/step

show 80 samples of predicted data.

In [32]:
import matplotlib.pyplot as plt

rand = np.random.randint(0,9999,(1,80))

plt.figure ( figsize = (12,15) )

for i in range(80):
    Sample = pre_X_test[rand[0][i]]
    plt.subplot(10,10,i+1)
    plt.imshow( Sample , cmap = "binary" )
    plt.axis("off")
    if Y_test_predict[rand[0][i]] == 0 :
        plt.title('T-shirt/top', fontsize = 12)
    elif Y_test_predict[rand[0][i]] == 1 :
        plt.title('Trouser', fontsize = 12)
    elif Y_test_predict[rand[0][i]] == 2 :
        plt.title('Pullover', fontsize = 12)
    elif Y_test_predict[rand[0][i]] == 3 :
        plt.title('Dress', fontsize = 12)
    elif Y_test_predict[rand[0][i]] == 4 :
        plt.title('Coat', fontsize = 12)
    elif Y_test_predict[rand[0][i]] == 5 :
        plt.title('Sandal', fontsize = 12)
    elif Y_test_predict[rand[0][i]] == 6 :
        plt.title('Shirt', fontsize = 12)
    elif Y_test_predict[rand[0][i]] == 7 :
        plt.title('Sneaker', fontsize = 12)
    elif Y_test_predict[rand[0][i]] == 8 :
        plt.title('Bag', fontsize = 12)
    elif Y_test_predict[rand[0][i]] == 9 :
        plt.title('Ankle boot', fontsize = 12)
    plt.show

calculate the confusion matrix

In [33]:
from sklearn.metrics import confusion_matrix

#Y_test_predict = Y_test_predict.astype('float32')

confusion_matrix ( pre_Y_test , Y_test_predict )
Out[33]:
array([[813,   8,  12,  62,   5,   4,  81,   0,  15,   0],
       [  8, 933,  15,  35,   7,   0,   0,   0,   2,   0],
       [ 23,   3, 690,   7, 180,   1,  84,   0,  12,   0],
       [ 40,  15,  13, 845,  35,   1,  46,   0,   5,   0],
       [  1,   2, 116,  41, 748,   0,  85,   0,   7,   0],
       [  0,   0,   0,   2,   0, 869,   0,  80,   2,  47],
       [206,   3, 138,  42, 125,   3, 444,   0,  39,   0],
       [  0,   0,   0,   0,   0,  33,   0, 879,   0,  88],
       [  2,   2,  18,  12,   2,   8,  18,   9, 928,   1],
       [  0,   0,   0,   1,   0,  13,   0,  50,   1, 935]], dtype=int64)

08.png

change the the number of neurons in each layer.

In [34]:
from keras.models import Sequential
from keras.layers import Dense
from keras.optimizers import SGD
from keras.losses import categorical_crossentropy

MLP_Model = Sequential()
MLP_Model.add(Dense(500, activation='relu', input_shape=(784,)))
MLP_Model.add(Dense(100, activation='relu'))
MLP_Model.add(Dense(10, activation='softmax'))

MLP_Model.summary()
MLP_Model.compile(optimizer = SGD(lr=0.001) , loss = categorical_crossentropy , metrics=['accuracy'])
Model: "sequential_4"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 dense_12 (Dense)            (None, 500)               392500    
                                                                 
 dense_13 (Dense)            (None, 100)               50100     
                                                                 
 dense_14 (Dense)            (None, 10)                1010      
                                                                 
=================================================================
Total params: 443,610
Trainable params: 443,610
Non-trainable params: 0
_________________________________________________________________
C:\Users\ali\anaconda3\lib\site-packages\keras\optimizers\optimizer_v2\gradient_descent.py:108: UserWarning: The `lr` argument is deprecated, use `learning_rate` instead.
  super(SGD, self).__init__(name, **kwargs)

observe the effect of the number of neurons on losses, val_losses, accuracy and val_accuracy.

In [35]:
network_history = MLP_Model.fit(X_train, Y_train, batch_size=32, epochs=25, validation_split=0.2)

history = network_history.history
import matplotlib.pyplot as plt
losses = history['loss']
val_losses = history['val_loss']
accuracies = history['accuracy']
val_accuracies = history['val_accuracy']
Epoch 1/25
1500/1500 [==============================] - 7s 4ms/step - loss: 1.4598 - accuracy: 0.5964 - val_loss: 0.9946 - val_accuracy: 0.7012
Epoch 2/25
1500/1500 [==============================] - 6s 4ms/step - loss: 0.8756 - accuracy: 0.7221 - val_loss: 0.7798 - val_accuracy: 0.7504
Epoch 3/25
1500/1500 [==============================] - 6s 4ms/step - loss: 0.7376 - accuracy: 0.7619 - val_loss: 0.6915 - val_accuracy: 0.7748
Epoch 4/25
1500/1500 [==============================] - 6s 4ms/step - loss: 0.6660 - accuracy: 0.7853 - val_loss: 0.6391 - val_accuracy: 0.7905
Epoch 5/25
1500/1500 [==============================] - 6s 4ms/step - loss: 0.6187 - accuracy: 0.8015 - val_loss: 0.5987 - val_accuracy: 0.8049
Epoch 6/25
1500/1500 [==============================] - 6s 4ms/step - loss: 0.5840 - accuracy: 0.8111 - val_loss: 0.5716 - val_accuracy: 0.8103
Epoch 7/25
1500/1500 [==============================] - 6s 4ms/step - loss: 0.5582 - accuracy: 0.8188 - val_loss: 0.5505 - val_accuracy: 0.8191
Epoch 8/25
1500/1500 [==============================] - 6s 4ms/step - loss: 0.5374 - accuracy: 0.8239 - val_loss: 0.5318 - val_accuracy: 0.8232
Epoch 9/25
1500/1500 [==============================] - 6s 4ms/step - loss: 0.5212 - accuracy: 0.8279 - val_loss: 0.5185 - val_accuracy: 0.8248
Epoch 10/25
1500/1500 [==============================] - 6s 4ms/step - loss: 0.5077 - accuracy: 0.8319 - val_loss: 0.5066 - val_accuracy: 0.8271
Epoch 11/25
1500/1500 [==============================] - 7s 5ms/step - loss: 0.4961 - accuracy: 0.8346 - val_loss: 0.4964 - val_accuracy: 0.8292
Epoch 12/25
1500/1500 [==============================] - 7s 4ms/step - loss: 0.4862 - accuracy: 0.8373 - val_loss: 0.4937 - val_accuracy: 0.8307
Epoch 13/25
1500/1500 [==============================] - 7s 5ms/step - loss: 0.4780 - accuracy: 0.8388 - val_loss: 0.4811 - val_accuracy: 0.8347
Epoch 14/25
1500/1500 [==============================] - 7s 4ms/step - loss: 0.4702 - accuracy: 0.8418 - val_loss: 0.4742 - val_accuracy: 0.8344
Epoch 15/25
1500/1500 [==============================] - 6s 4ms/step - loss: 0.4636 - accuracy: 0.8440 - val_loss: 0.4683 - val_accuracy: 0.8375
Epoch 16/25
1500/1500 [==============================] - 6s 4ms/step - loss: 0.4571 - accuracy: 0.8456 - val_loss: 0.4664 - val_accuracy: 0.8352
Epoch 17/25
1500/1500 [==============================] - 6s 4ms/step - loss: 0.4511 - accuracy: 0.8465 - val_loss: 0.4591 - val_accuracy: 0.8411
Epoch 18/25
1500/1500 [==============================] - 6s 4ms/step - loss: 0.4463 - accuracy: 0.8483 - val_loss: 0.4540 - val_accuracy: 0.8417
Epoch 19/25
1500/1500 [==============================] - 6s 4ms/step - loss: 0.4417 - accuracy: 0.8488 - val_loss: 0.4489 - val_accuracy: 0.8432
Epoch 20/25
1500/1500 [==============================] - 7s 4ms/step - loss: 0.4373 - accuracy: 0.8512 - val_loss: 0.4496 - val_accuracy: 0.8415
Epoch 21/25
1500/1500 [==============================] - 6s 4ms/step - loss: 0.4331 - accuracy: 0.8513 - val_loss: 0.4443 - val_accuracy: 0.8447
Epoch 22/25
1500/1500 [==============================] - 6s 4ms/step - loss: 0.4292 - accuracy: 0.8531 - val_loss: 0.4388 - val_accuracy: 0.8459
Epoch 23/25
1500/1500 [==============================] - 6s 4ms/step - loss: 0.4259 - accuracy: 0.8549 - val_loss: 0.4361 - val_accuracy: 0.8461
Epoch 24/25
1500/1500 [==============================] - 7s 5ms/step - loss: 0.4224 - accuracy: 0.8554 - val_loss: 0.4332 - val_accuracy: 0.8489
Epoch 25/25
1500/1500 [==============================] - 7s 5ms/step - loss: 0.4188 - accuracy: 0.8571 - val_loss: 0.4293 - val_accuracy: 0.8497

show the fact that loss and val_loss decrease by each epoch

In [36]:
plt.xlabel('Epochs')
plt.ylabel('Loss')
plt.plot(losses)
plt.plot(val_losses)
plt.legend(['loss', 'val_loss'])
Out[36]:
<matplotlib.legend.Legend at 0x25cd500a708>

show the fact that accuray and val_accuray increase by each epoch.

In [37]:
plt.xlabel('Epochs')
plt.ylabel('Accuracy')
plt.plot(accuracies)
plt.plot(val_accuracies)
plt.legend(['acc', 'val_acc'])
Out[37]:
<matplotlib.legend.Legend at 0x25cd3883608>

run the model on test data and evaluate the loss and accuracy

In [38]:
test_loss, test_acc = MLP_Model.evaluate(X_test, Y_test)

Y_test_predict = MLP_Model.predict(X_test)

Y_test_predict = np.argmax(Y_test_predict, axis=1)
313/313 [==============================] - 1s 3ms/step - loss: 0.4526 - accuracy: 0.8435
313/313 [==============================] - 1s 2ms/step

show 80 samples of predicted data.

In [39]:
import matplotlib.pyplot as plt

rand = np.random.randint(0,9999,(1,80))

plt.figure ( figsize = (12,15) )

for i in range(80):
    Sample = pre_X_test[rand[0][i]]
    plt.subplot(10,10,i+1)
    plt.imshow( Sample , cmap = "binary" )
    plt.axis("off")
    if Y_test_predict[rand[0][i]] == 0 :
        plt.title('T-shirt/top', fontsize = 12)
    elif Y_test_predict[rand[0][i]] == 1 :
        plt.title('Trouser', fontsize = 12)
    elif Y_test_predict[rand[0][i]] == 2 :
        plt.title('Pullover', fontsize = 12)
    elif Y_test_predict[rand[0][i]] == 3 :
        plt.title('Dress', fontsize = 12)
    elif Y_test_predict[rand[0][i]] == 4 :
        plt.title('Coat', fontsize = 12)
    elif Y_test_predict[rand[0][i]] == 5 :
        plt.title('Sandal', fontsize = 12)
    elif Y_test_predict[rand[0][i]] == 6 :
        plt.title('Shirt', fontsize = 12)
    elif Y_test_predict[rand[0][i]] == 7 :
        plt.title('Sneaker', fontsize = 12)
    elif Y_test_predict[rand[0][i]] == 8 :
        plt.title('Bag', fontsize = 12)
    elif Y_test_predict[rand[0][i]] == 9 :
        plt.title('Ankle boot', fontsize = 12)
    plt.show

calculate the confusion matrix

In [40]:
from sklearn.metrics import confusion_matrix

#Y_test_predict = Y_test_predict.astype('float32')

confusion_matrix ( pre_Y_test , Y_test_predict )
Out[40]:
array([[800,   4,  17,  56,   7,   2,  98,   0,  16,   0],
       [  3, 948,  10,  30,   6,   0,   1,   0,   2,   0],
       [ 17,   3, 759,  11, 130,   1,  71,   0,   8,   0],
       [ 25,  13,  14, 865,  36,   0,  42,   0,   5,   0],
       [  0,   2, 108,  33, 771,   1,  78,   0,   7,   0],
       [  0,   0,   0,   1,   0, 919,   0,  49,   2,  29],
       [136,   1, 116,  40, 102,   1, 576,   0,  28,   0],
       [  0,   0,   0,   0,   0,  39,   0, 905,   0,  56],
       [  2,   1,  14,   9,   3,   5,  16,   5, 945,   0],
       [  0,   0,   0,   0,   0,  17,   0,  35,   1, 947]], dtype=int64)

change the the number of neuron in the first layer(1000 to 500).

In [41]:
from keras.models import Sequential
from keras.layers import Dense
from keras.optimizers import SGD
from keras.losses import categorical_crossentropy

MLP_Model = Sequential()
MLP_Model.add(Dense(500, activation='relu', input_shape=(784,)))
MLP_Model.add(Dense(100, activation='relu'))
MLP_Model.add(Dense(10, activation='softmax'))

MLP_Model.summary()
MLP_Model.compile(optimizer = SGD(lr=0.001) , loss = categorical_crossentropy , metrics=['accuracy'])
Model: "sequential_5"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 dense_15 (Dense)            (None, 500)               392500    
                                                                 
 dense_16 (Dense)            (None, 100)               50100     
                                                                 
 dense_17 (Dense)            (None, 10)                1010      
                                                                 
=================================================================
Total params: 443,610
Trainable params: 443,610
Non-trainable params: 0
_________________________________________________________________
C:\Users\ali\anaconda3\lib\site-packages\keras\optimizers\optimizer_v2\gradient_descent.py:108: UserWarning: The `lr` argument is deprecated, use `learning_rate` instead.
  super(SGD, self).__init__(name, **kwargs)

observe the effect of the number of neurons on losses, val_losses, accuracy and val_accuracy.

In [42]:
network_history = MLP_Model.fit(X_train, Y_train, batch_size=64, epochs=25, validation_split=0.2)

history = network_history.history
import matplotlib.pyplot as plt
losses = history['loss']
val_losses = history['val_loss']
accuracies = history['accuracy']
val_accuracies = history['val_accuracy']
Epoch 1/25
750/750 [==============================] - 4s 5ms/step - loss: 1.7265 - accuracy: 0.5060 - val_loss: 1.3159 - val_accuracy: 0.6535
Epoch 2/25
750/750 [==============================] - 3s 4ms/step - loss: 1.1357 - accuracy: 0.6739 - val_loss: 0.9828 - val_accuracy: 0.7024
Epoch 3/25
750/750 [==============================] - 3s 4ms/step - loss: 0.9143 - accuracy: 0.7166 - val_loss: 0.8396 - val_accuracy: 0.7345
Epoch 4/25
750/750 [==============================] - 3s 5ms/step - loss: 0.8063 - accuracy: 0.7460 - val_loss: 0.7608 - val_accuracy: 0.7573
Epoch 5/25
750/750 [==============================] - 3s 4ms/step - loss: 0.7408 - accuracy: 0.7645 - val_loss: 0.7089 - val_accuracy: 0.7736
Epoch 6/25
750/750 [==============================] - 4s 5ms/step - loss: 0.6955 - accuracy: 0.7769 - val_loss: 0.6714 - val_accuracy: 0.7832
Epoch 7/25
750/750 [==============================] - 3s 4ms/step - loss: 0.6610 - accuracy: 0.7866 - val_loss: 0.6421 - val_accuracy: 0.7908
Epoch 8/25
750/750 [==============================] - 3s 5ms/step - loss: 0.6339 - accuracy: 0.7950 - val_loss: 0.6186 - val_accuracy: 0.7977
Epoch 9/25
750/750 [==============================] - 3s 5ms/step - loss: 0.6119 - accuracy: 0.8008 - val_loss: 0.6000 - val_accuracy: 0.8023
Epoch 10/25
750/750 [==============================] - 3s 4ms/step - loss: 0.5937 - accuracy: 0.8061 - val_loss: 0.5841 - val_accuracy: 0.8074
Epoch 11/25
750/750 [==============================] - 3s 5ms/step - loss: 0.5779 - accuracy: 0.8114 - val_loss: 0.5727 - val_accuracy: 0.8109
Epoch 12/25
750/750 [==============================] - 3s 4ms/step - loss: 0.5642 - accuracy: 0.8139 - val_loss: 0.5594 - val_accuracy: 0.8146
Epoch 13/25
750/750 [==============================] - 3s 4ms/step - loss: 0.5526 - accuracy: 0.8175 - val_loss: 0.5485 - val_accuracy: 0.8168
Epoch 14/25
750/750 [==============================] - 3s 4ms/step - loss: 0.5421 - accuracy: 0.8209 - val_loss: 0.5405 - val_accuracy: 0.8185
Epoch 15/25
750/750 [==============================] - 3s 5ms/step - loss: 0.5327 - accuracy: 0.8234 - val_loss: 0.5339 - val_accuracy: 0.8192
Epoch 16/25
750/750 [==============================] - 3s 5ms/step - loss: 0.5245 - accuracy: 0.8257 - val_loss: 0.5239 - val_accuracy: 0.8213
Epoch 17/25
750/750 [==============================] - 3s 4ms/step - loss: 0.5169 - accuracy: 0.8274 - val_loss: 0.5172 - val_accuracy: 0.8237
Epoch 18/25
750/750 [==============================] - 3s 4ms/step - loss: 0.5101 - accuracy: 0.8287 - val_loss: 0.5112 - val_accuracy: 0.8242
Epoch 19/25
750/750 [==============================] - 3s 5ms/step - loss: 0.5038 - accuracy: 0.8310 - val_loss: 0.5046 - val_accuracy: 0.8271
Epoch 20/25
750/750 [==============================] - 3s 4ms/step - loss: 0.4981 - accuracy: 0.8319 - val_loss: 0.5000 - val_accuracy: 0.8274
Epoch 21/25
750/750 [==============================] - 3s 4ms/step - loss: 0.4926 - accuracy: 0.8339 - val_loss: 0.4948 - val_accuracy: 0.8293
Epoch 22/25
750/750 [==============================] - 3s 5ms/step - loss: 0.4877 - accuracy: 0.8356 - val_loss: 0.4901 - val_accuracy: 0.8307
Epoch 23/25
750/750 [==============================] - 4s 5ms/step - loss: 0.4831 - accuracy: 0.8369 - val_loss: 0.4862 - val_accuracy: 0.8325
Epoch 24/25
750/750 [==============================] - 3s 5ms/step - loss: 0.4787 - accuracy: 0.8380 - val_loss: 0.4823 - val_accuracy: 0.8335
Epoch 25/25
750/750 [==============================] - 3s 4ms/step - loss: 0.4747 - accuracy: 0.8396 - val_loss: 0.4808 - val_accuracy: 0.8333

show the fact that loss and val_loss decrease by each epoch

In [43]:
plt.xlabel('Epochs')
plt.ylabel('Loss')
plt.plot(losses)
plt.plot(val_losses)
plt.legend(['loss', 'val_loss'])
Out[43]:
<matplotlib.legend.Legend at 0x25cd476fa48>

show the fact that accuray and val_accuray increase by each epoch.

In [44]:
plt.xlabel('Epochs')
plt.ylabel('Accuracy')
plt.plot(accuracies)
plt.plot(val_accuracies)
plt.legend(['acc', 'val_acc'])
Out[44]:
<matplotlib.legend.Legend at 0x25cd46b81c8>

run the model on test data and evaluate the loss and accuracy

In [45]:
test_loss, test_acc = MLP_Model.evaluate(X_test, Y_test)

Y_test_predict = MLP_Model.predict(X_test)

Y_test_predict = np.argmax(Y_test_predict, axis=1)
313/313 [==============================] - 1s 3ms/step - loss: 0.5040 - accuracy: 0.8237
313/313 [==============================] - 1s 2ms/step

show 80 samples of predicted data.

In [46]:
import matplotlib.pyplot as plt

rand = np.random.randint(0,9999,(1,80))

plt.figure ( figsize = (12,15) )

for i in range(80):
    Sample = pre_X_test[rand[0][i]]
    plt.subplot(10,10,i+1)
    plt.imshow( Sample , cmap = "binary" )
    plt.axis("off")
    if Y_test_predict[rand[0][i]] == 0 :
        plt.title('T-shirt/top', fontsize = 12)
    elif Y_test_predict[rand[0][i]] == 1 :
        plt.title('Trouser', fontsize = 12)
    elif Y_test_predict[rand[0][i]] == 2 :
        plt.title('Pullover', fontsize = 12)
    elif Y_test_predict[rand[0][i]] == 3 :
        plt.title('Dress', fontsize = 12)
    elif Y_test_predict[rand[0][i]] == 4 :
        plt.title('Coat', fontsize = 12)
    elif Y_test_predict[rand[0][i]] == 5 :
        plt.title('Sandal', fontsize = 12)
    elif Y_test_predict[rand[0][i]] == 6 :
        plt.title('Shirt', fontsize = 12)
    elif Y_test_predict[rand[0][i]] == 7 :
        plt.title('Sneaker', fontsize = 12)
    elif Y_test_predict[rand[0][i]] == 8 :
        plt.title('Bag', fontsize = 12)
    elif Y_test_predict[rand[0][i]] == 9 :
        plt.title('Ankle boot', fontsize = 12)
    plt.show

calculate the confusion matrix

In [47]:
from sklearn.metrics import confusion_matrix

#Y_test_predict = Y_test_predict.astype('float32')

confusion_matrix ( pre_Y_test , Y_test_predict )
Out[47]:
array([[763,   6,  15,  59,   7,   1, 130,   0,  19,   0],
       [  5, 936,   9,  37,   7,   0,   4,   0,   2,   0],
       [ 18,   4, 730,  10, 120,   1, 108,   0,   9,   0],
       [ 28,  13,  12, 864,  29,   0,  47,   0,   7,   0],
       [  0,   3, 125,  35, 714,   1, 116,   0,   6,   0],
       [  0,   0,   0,   1,   0, 896,   0,  56,   3,  44],
       [128,   2, 130,  43,  89,   0, 577,   0,  31,   0],
       [  0,   0,   0,   0,   0,  39,   0, 892,   0,  69],
       [  0,   1,  13,  10,   1,   8,  27,   6, 933,   1],
       [  0,   0,   0,   0,   0,  15,   1,  51,   1, 932]], dtype=int64)

change the the number of neurons.

In [48]:
from keras.models import Sequential
from keras.layers import Dense
from keras.optimizers import SGD
from keras.losses import categorical_crossentropy

MLP_Model = Sequential()
MLP_Model.add(Dense(500, activation='relu', input_shape=(784,)))
MLP_Model.add(Dense(100, activation='relu'))
MLP_Model.add(Dense(10, activation='softmax'))

MLP_Model.summary()
MLP_Model.compile(optimizer = SGD(lr=0.001) , loss = categorical_crossentropy , metrics=['accuracy'])
Model: "sequential_6"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 dense_18 (Dense)            (None, 500)               392500    
                                                                 
 dense_19 (Dense)            (None, 100)               50100     
                                                                 
 dense_20 (Dense)            (None, 10)                1010      
                                                                 
=================================================================
Total params: 443,610
Trainable params: 443,610
Non-trainable params: 0
_________________________________________________________________
C:\Users\ali\anaconda3\lib\site-packages\keras\optimizers\optimizer_v2\gradient_descent.py:108: UserWarning: The `lr` argument is deprecated, use `learning_rate` instead.
  super(SGD, self).__init__(name, **kwargs)

observe the effect of the number of neurons on losses, val_losses, accuracy and val_accuracy.

In [49]:
network_history = MLP_Model.fit(X_train, Y_train, batch_size=256, epochs=25, validation_split=0.2)

history = network_history.history
import matplotlib.pyplot as plt
losses = history['loss']
val_losses = history['val_loss']
accuracies = history['accuracy']
val_accuracies = history['val_accuracy']
Epoch 1/25
188/188 [==============================] - 2s 8ms/step - loss: 2.0909 - accuracy: 0.3104 - val_loss: 1.8989 - val_accuracy: 0.4665
Epoch 2/25
188/188 [==============================] - 1s 6ms/step - loss: 1.7671 - accuracy: 0.5374 - val_loss: 1.6377 - val_accuracy: 0.5923
Epoch 3/25
188/188 [==============================] - 1s 7ms/step - loss: 1.5461 - accuracy: 0.6049 - val_loss: 1.4476 - val_accuracy: 0.6209
Epoch 4/25
188/188 [==============================] - 1s 7ms/step - loss: 1.3820 - accuracy: 0.6244 - val_loss: 1.3045 - val_accuracy: 0.6404
Epoch 5/25
188/188 [==============================] - 1s 6ms/step - loss: 1.2575 - accuracy: 0.6427 - val_loss: 1.1956 - val_accuracy: 0.6545
Epoch 6/25
188/188 [==============================] - 1s 6ms/step - loss: 1.1618 - accuracy: 0.6537 - val_loss: 1.1112 - val_accuracy: 0.6705
Epoch 7/25
188/188 [==============================] - 1s 7ms/step - loss: 1.0869 - accuracy: 0.6688 - val_loss: 1.0444 - val_accuracy: 0.6805
Epoch 8/25
188/188 [==============================] - 1s 6ms/step - loss: 1.0272 - accuracy: 0.6812 - val_loss: 0.9910 - val_accuracy: 0.6950
Epoch 9/25
188/188 [==============================] - 1s 7ms/step - loss: 0.9787 - accuracy: 0.6934 - val_loss: 0.9472 - val_accuracy: 0.7068
Epoch 10/25
188/188 [==============================] - 1s 7ms/step - loss: 0.9385 - accuracy: 0.7039 - val_loss: 0.9113 - val_accuracy: 0.7168
Epoch 11/25
188/188 [==============================] - 1s 7ms/step - loss: 0.9049 - accuracy: 0.7143 - val_loss: 0.8799 - val_accuracy: 0.7229
Epoch 12/25
188/188 [==============================] - 1s 7ms/step - loss: 0.8762 - accuracy: 0.7224 - val_loss: 0.8536 - val_accuracy: 0.7322
Epoch 13/25
188/188 [==============================] - 1s 7ms/step - loss: 0.8515 - accuracy: 0.7297 - val_loss: 0.8308 - val_accuracy: 0.7395
Epoch 14/25
188/188 [==============================] - 1s 6ms/step - loss: 0.8297 - accuracy: 0.7362 - val_loss: 0.8109 - val_accuracy: 0.7449
Epoch 15/25
188/188 [==============================] - 1s 7ms/step - loss: 0.8104 - accuracy: 0.7416 - val_loss: 0.7926 - val_accuracy: 0.7499
Epoch 16/25
188/188 [==============================] - 1s 7ms/step - loss: 0.7930 - accuracy: 0.7462 - val_loss: 0.7766 - val_accuracy: 0.7559
Epoch 17/25
188/188 [==============================] - 1s 7ms/step - loss: 0.7774 - accuracy: 0.7517 - val_loss: 0.7617 - val_accuracy: 0.7598
Epoch 18/25
188/188 [==============================] - 1s 6ms/step - loss: 0.7630 - accuracy: 0.7555 - val_loss: 0.7484 - val_accuracy: 0.7615
Epoch 19/25
188/188 [==============================] - 1s 7ms/step - loss: 0.7499 - accuracy: 0.7600 - val_loss: 0.7363 - val_accuracy: 0.7650
Epoch 20/25
188/188 [==============================] - 1s 7ms/step - loss: 0.7377 - accuracy: 0.7637 - val_loss: 0.7246 - val_accuracy: 0.7673
Epoch 21/25
188/188 [==============================] - 1s 7ms/step - loss: 0.7265 - accuracy: 0.7670 - val_loss: 0.7139 - val_accuracy: 0.7697
Epoch 22/25
188/188 [==============================] - 1s 7ms/step - loss: 0.7159 - accuracy: 0.7704 - val_loss: 0.7043 - val_accuracy: 0.7737
Epoch 23/25
188/188 [==============================] - 1s 7ms/step - loss: 0.7062 - accuracy: 0.7727 - val_loss: 0.6950 - val_accuracy: 0.7756
Epoch 24/25
188/188 [==============================] - 1s 7ms/step - loss: 0.6969 - accuracy: 0.7761 - val_loss: 0.6862 - val_accuracy: 0.7773
Epoch 25/25
188/188 [==============================] - 1s 7ms/step - loss: 0.6883 - accuracy: 0.7788 - val_loss: 0.6785 - val_accuracy: 0.7797

show the fact that loss and val_loss decrease by each epoch

In [50]:
plt.xlabel('Epochs')
plt.ylabel('Loss')
plt.plot(losses)
plt.plot(val_losses)
plt.legend(['loss', 'val_loss'])
Out[50]:
<matplotlib.legend.Legend at 0x25cd6b06b48>

show the fact that accuray and val_accuray increase by each epoch.

In [51]:
plt.xlabel('Epochs')
plt.ylabel('Accuracy')
plt.plot(accuracies)
plt.plot(val_accuracies)
plt.legend(['acc', 'val_acc'])
Out[51]:
<matplotlib.legend.Legend at 0x25cd382a408>

run the model on test data and evaluate the loss and accuracy

In [52]:
test_loss, test_acc = MLP_Model.evaluate(X_test, Y_test)

Y_test_predict = MLP_Model.predict(X_test)

Y_test_predict = np.argmax(Y_test_predict, axis=1)
313/313 [==============================] - 1s 3ms/step - loss: 0.7020 - accuracy: 0.7658
313/313 [==============================] - 1s 2ms/step

show 80 samples of predicted data.

In [53]:
import matplotlib.pyplot as plt

rand = np.random.randint(0,9999,(1,80))

plt.figure ( figsize = (12,15) )

for i in range(80):
    Sample = pre_X_test[rand[0][i]]
    plt.subplot(10,10,i+1)
    plt.imshow( Sample , cmap = "binary" )
    plt.axis("off")
    if Y_test_predict[rand[0][i]] == 0 :
        plt.title('T-shirt/top', fontsize = 12)
    elif Y_test_predict[rand[0][i]] == 1 :
        plt.title('Trouser', fontsize = 12)
    elif Y_test_predict[rand[0][i]] == 2 :
        plt.title('Pullover', fontsize = 12)
    elif Y_test_predict[rand[0][i]] == 3 :
        plt.title('Dress', fontsize = 12)
    elif Y_test_predict[rand[0][i]] == 4 :
        plt.title('Coat', fontsize = 12)
    elif Y_test_predict[rand[0][i]] == 5 :
        plt.title('Sandal', fontsize = 12)
    elif Y_test_predict[rand[0][i]] == 6 :
        plt.title('Shirt', fontsize = 12)
    elif Y_test_predict[rand[0][i]] == 7 :
        plt.title('Sneaker', fontsize = 12)
    elif Y_test_predict[rand[0][i]] == 8 :
        plt.title('Bag', fontsize = 12)
    elif Y_test_predict[rand[0][i]] == 9 :
        plt.title('Ankle boot', fontsize = 12)
    plt.show

calculate the confusion matrix

In [54]:
from sklearn.metrics import confusion_matrix

#Y_test_predict = Y_test_predict.astype('float32')

confusion_matrix ( pre_Y_test , Y_test_predict )
Out[54]:
array([[772,   5,   8,  95,   6,   0,  87,   0,  26,   1],
       [ 13, 914,  12,  45,  13,   0,   1,   0,   2,   0],
       [ 17,   1, 586,  10, 253,   0, 111,   0,  21,   1],
       [ 33,  13,   1, 840,  38,   1,  67,   0,   7,   0],
       [  3,   3, 148,  45, 727,   0,  66,   0,   8,   0],
       [  1,   0,   0,   2,   0, 708,   1, 185,  10,  93],
       [210,   5, 124,  54, 201,   0, 360,   0,  46,   0],
       [  0,   0,   0,   0,   0,  29,   0, 886,   0,  85],
       [  5,   1,  11,  13,   1,  12,  17,  11, 927,   2],
       [  0,   0,   0,   2,   0,   2,   0,  57,   1, 938]], dtype=int64)

09.png

change the the number of neurons.

In [55]:
from keras.models import Sequential
from keras.layers import Dense
from keras.optimizers import SGD
from keras.losses import categorical_crossentropy

MLP_Model = Sequential()
MLP_Model.add(Dense(1000, activation='relu', input_shape=(784,)))
MLP_Model.add(Dense(500, activation='relu'))
MLP_Model.add(Dense(10, activation='softmax'))

MLP_Model.summary()
MLP_Model.compile(optimizer = SGD(lr=0.001) , loss = categorical_crossentropy , metrics=['accuracy'])
Model: "sequential_7"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 dense_21 (Dense)            (None, 1000)              785000    
                                                                 
 dense_22 (Dense)            (None, 500)               500500    
                                                                 
 dense_23 (Dense)            (None, 10)                5010      
                                                                 
=================================================================
Total params: 1,290,510
Trainable params: 1,290,510
Non-trainable params: 0
_________________________________________________________________
C:\Users\ali\anaconda3\lib\site-packages\keras\optimizers\optimizer_v2\gradient_descent.py:108: UserWarning: The `lr` argument is deprecated, use `learning_rate` instead.
  super(SGD, self).__init__(name, **kwargs)

observe the effect of the number of neurons on losses, val_losses, accuracy and val_accuracy.

In [56]:
network_history = MLP_Model.fit(X_train, Y_train, batch_size=32, epochs=30, validation_split=0.2)

history = network_history.history
import matplotlib.pyplot as plt
losses = history['loss']
val_losses = history['val_loss']
accuracies = history['accuracy']
val_accuracies = history['val_accuracy']
Epoch 1/30
1500/1500 [==============================] - 14s 9ms/step - loss: 1.3599 - accuracy: 0.6266 - val_loss: 0.9364 - val_accuracy: 0.7193
Epoch 2/30
1500/1500 [==============================] - 14s 9ms/step - loss: 0.8290 - accuracy: 0.7446 - val_loss: 0.7409 - val_accuracy: 0.7671
Epoch 3/30
1500/1500 [==============================] - 14s 9ms/step - loss: 0.7012 - accuracy: 0.7786 - val_loss: 0.6570 - val_accuracy: 0.7884
Epoch 4/30
1500/1500 [==============================] - 16s 10ms/step - loss: 0.6344 - accuracy: 0.7995 - val_loss: 0.6072 - val_accuracy: 0.8045
Epoch 5/30
1500/1500 [==============================] - 16s 10ms/step - loss: 0.5913 - accuracy: 0.8097 - val_loss: 0.5733 - val_accuracy: 0.8152
Epoch 6/30
1500/1500 [==============================] - 16s 10ms/step - loss: 0.5604 - accuracy: 0.8176 - val_loss: 0.5478 - val_accuracy: 0.8178
Epoch 7/30
1500/1500 [==============================] - 15s 10ms/step - loss: 0.5371 - accuracy: 0.8238 - val_loss: 0.5318 - val_accuracy: 0.8224
Epoch 8/30
1500/1500 [==============================] - 15s 10ms/step - loss: 0.5189 - accuracy: 0.8279 - val_loss: 0.5146 - val_accuracy: 0.8277
Epoch 9/30
1500/1500 [==============================] - 16s 10ms/step - loss: 0.5042 - accuracy: 0.8328 - val_loss: 0.5025 - val_accuracy: 0.8307
Epoch 10/30
1500/1500 [==============================] - 16s 11ms/step - loss: 0.4920 - accuracy: 0.8354 - val_loss: 0.4912 - val_accuracy: 0.8317
Epoch 11/30
1500/1500 [==============================] - 16s 10ms/step - loss: 0.4818 - accuracy: 0.8380 - val_loss: 0.4831 - val_accuracy: 0.8341
Epoch 12/30
1500/1500 [==============================] - 15s 10ms/step - loss: 0.4726 - accuracy: 0.8413 - val_loss: 0.4749 - val_accuracy: 0.8357
Epoch 13/30
1500/1500 [==============================] - 15s 10ms/step - loss: 0.4645 - accuracy: 0.8430 - val_loss: 0.4702 - val_accuracy: 0.8375
Epoch 14/30
1500/1500 [==============================] - 14s 10ms/step - loss: 0.4578 - accuracy: 0.8450 - val_loss: 0.4615 - val_accuracy: 0.8404
Epoch 15/30
1500/1500 [==============================] - 15s 10ms/step - loss: 0.4510 - accuracy: 0.8465 - val_loss: 0.4579 - val_accuracy: 0.8417
Epoch 16/30
1500/1500 [==============================] - 15s 10ms/step - loss: 0.4452 - accuracy: 0.8480 - val_loss: 0.4535 - val_accuracy: 0.8407
Epoch 17/30
1500/1500 [==============================] - 15s 10ms/step - loss: 0.4401 - accuracy: 0.8502 - val_loss: 0.4461 - val_accuracy: 0.8452
Epoch 18/30
1500/1500 [==============================] - 15s 10ms/step - loss: 0.4353 - accuracy: 0.8512 - val_loss: 0.4418 - val_accuracy: 0.8445
Epoch 19/30
1500/1500 [==============================] - 17s 11ms/step - loss: 0.4301 - accuracy: 0.8530 - val_loss: 0.4387 - val_accuracy: 0.8473
Epoch 20/30
1500/1500 [==============================] - 17s 12ms/step - loss: 0.4263 - accuracy: 0.8539 - val_loss: 0.4361 - val_accuracy: 0.8475
Epoch 21/30
1500/1500 [==============================] - 16s 11ms/step - loss: 0.4223 - accuracy: 0.8554 - val_loss: 0.4323 - val_accuracy: 0.8474
Epoch 22/30
1500/1500 [==============================] - 16s 11ms/step - loss: 0.4187 - accuracy: 0.8567 - val_loss: 0.4283 - val_accuracy: 0.8507
Epoch 23/30
1500/1500 [==============================] - 17s 11ms/step - loss: 0.4150 - accuracy: 0.8576 - val_loss: 0.4259 - val_accuracy: 0.8518
Epoch 24/30
1500/1500 [==============================] - 17s 11ms/step - loss: 0.4112 - accuracy: 0.8584 - val_loss: 0.4239 - val_accuracy: 0.8522
Epoch 25/30
1500/1500 [==============================] - 18s 12ms/step - loss: 0.4083 - accuracy: 0.8601 - val_loss: 0.4219 - val_accuracy: 0.8516
Epoch 26/30
1500/1500 [==============================] - 17s 12ms/step - loss: 0.4052 - accuracy: 0.8610 - val_loss: 0.4216 - val_accuracy: 0.8522
Epoch 27/30
1500/1500 [==============================] - 17s 11ms/step - loss: 0.4023 - accuracy: 0.8618 - val_loss: 0.4158 - val_accuracy: 0.8547
Epoch 28/30
1500/1500 [==============================] - 16s 11ms/step - loss: 0.3996 - accuracy: 0.8630 - val_loss: 0.4140 - val_accuracy: 0.8539
Epoch 29/30
1500/1500 [==============================] - 16s 10ms/step - loss: 0.3964 - accuracy: 0.8638 - val_loss: 0.4130 - val_accuracy: 0.8569
Epoch 30/30
1500/1500 [==============================] - 16s 10ms/step - loss: 0.3939 - accuracy: 0.8653 - val_loss: 0.4074 - val_accuracy: 0.8584

show the fact that loss and val_loss decrease by each epoch

In [57]:
plt.xlabel('Epochs')
plt.ylabel('Loss')
plt.plot(losses)
plt.plot(val_losses)
plt.legend(['loss', 'val_loss'])
Out[57]:
<matplotlib.legend.Legend at 0x25cd4943c48>

show the fact that accuray and val_accuray increase by each epoch.

In [58]:
plt.xlabel('Epochs')
plt.ylabel('Accuracy')
plt.plot(accuracies)
plt.plot(val_accuracies)
plt.legend(['acc', 'val_acc'])
Out[58]:
<matplotlib.legend.Legend at 0x25cd39e4448>

run the model on test data and evaluate the loss and accuracy

In [59]:
test_loss, test_acc = MLP_Model.evaluate(X_test, Y_test)

Y_test_predict = MLP_Model.predict(X_test)

Y_test_predict = np.argmax(Y_test_predict, axis=1)
313/313 [==============================] - 2s 5ms/step - loss: 0.4309 - accuracy: 0.8477
313/313 [==============================] - 2s 5ms/step

show 80 samples of predicted data.

In [60]:
import matplotlib.pyplot as plt

rand = np.random.randint(0,9999,(1,80))

plt.figure ( figsize = (12,15) )

for i in range(80):
    Sample = pre_X_test[rand[0][i]]
    plt.subplot(10,10,i+1)
    plt.imshow( Sample , cmap = "binary" )
    plt.axis("off")
    if Y_test_predict[rand[0][i]] == 0 :
        plt.title('T-shirt/top', fontsize = 12)
    elif Y_test_predict[rand[0][i]] == 1 :
        plt.title('Trouser', fontsize = 12)
    elif Y_test_predict[rand[0][i]] == 2 :
        plt.title('Pullover', fontsize = 12)
    elif Y_test_predict[rand[0][i]] == 3 :
        plt.title('Dress', fontsize = 12)
    elif Y_test_predict[rand[0][i]] == 4 :
        plt.title('Coat', fontsize = 12)
    elif Y_test_predict[rand[0][i]] == 5 :
        plt.title('Sandal', fontsize = 12)
    elif Y_test_predict[rand[0][i]] == 6 :
        plt.title('Shirt', fontsize = 12)
    elif Y_test_predict[rand[0][i]] == 7 :
        plt.title('Sneaker', fontsize = 12)
    elif Y_test_predict[rand[0][i]] == 8 :
        plt.title('Bag', fontsize = 12)
    elif Y_test_predict[rand[0][i]] == 9 :
        plt.title('Ankle boot', fontsize = 12)
    plt.show

calculate the confusion matrix

In [61]:
from sklearn.metrics import confusion_matrix

#Y_test_predict = Y_test_predict.astype('float32')

confusion_matrix ( pre_Y_test , Y_test_predict )
Out[61]:
array([[824,   3,  12,  44,   5,   4,  94,   0,  14,   0],
       [  4, 957,   2,  28,   4,   0,   4,   0,   1,   0],
       [ 19,   5, 731,   9, 139,   1,  85,   0,  11,   0],
       [ 31,  12,  12, 866,  30,   0,  44,   0,   5,   0],
       [  1,   2,  99,  39, 773,   1,  80,   0,   5,   0],
       [  0,   0,   0,   1,   0, 927,   0,  47,   1,  24],
       [154,   2, 101,  34,  94,   1, 589,   0,  25,   0],
       [  0,   0,   0,   0,   0,  34,   0, 916,   0,  50],
       [  3,   1,   7,   7,   4,   3,  18,   5, 952,   0],
       [  0,   0,   0,   0,   0,  16,   1,  40,   1, 942]], dtype=int64)